Skip to content

Refactor shared Pipe, consensus, and subscription helpers; cache LOAD models - #18837

Draft
Caideyipi wants to merge 4 commits into
apache:masterfrom
Caideyipi:refactor/align-shared-pipe-consensus-subscription
Draft

Caideyipi wants to merge 4 commits into
apache:masterfrom
Caideyipi:refactor/align-shared-pipe-consensus-subscription

Conversation

@Caideyipi

@Caideyipi Caideyipi commented Oct 12, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Refactors shared Pipe, IoTConsensus, and subscription code into focused helpers while retaining existing behavior. Adds an indexed model cache for TSFile LOAD analysis so mini-file conversion and tablet fallback dispatch by each file's actual model.

For input [tree-a.tsfile, table-a.tsfile, tree-b.tsfile], the cache retains model flags [false, true, false], and tablet fallback dispatches tree/table/tree converters. The reliable index advances only after each file's model metadata is read and classified.

  • Introduce PipeBatchMetricsSettable and extract batch detachment while preserving synchronization and atomic ownership handoff.
  • Extract tree-pattern splitting, modification-map construction, schema-snapshot batch flushing, and ConfigNode snapshot file-list helpers.
  • Extract consensus retry, missing-group response, and snapshot-cleanup helpers without changing retry decisions, status codes, diagnostics, or locking.
  • Route subscription tablet projection through TableViewTabletProjectorProvider; keep projector lifetime scoped to each batch and preserve topic configuration snapshots, TAG filtering, column pruning, and WAL replay.
  • Add regression coverage for consensus failure paths, subscription projector lifetime and configuration, TAG filtering, and LOAD model-cache index behavior.

Validation

  • The earlier shared-helper revision passed 248 focused unit tests and full-reactor package builds for the default and Chinese locales; Spotless, Checkstyle, and diff checks passed.
  • After the LOAD cache addition and this master sync, git diff --check passes. LoadTsFileModelCacheTest and test-compile remain unverified because the external test volume was unavailable; no test data or temporary files were redirected to the system drive.

@Caideyipi
Caideyipi force-pushed the refactor/align-shared-pipe-consensus-subscription branch from 7a22815 to 5cb1d22 Compare October 26, 2026 10:40
@Caideyipi

Copy link
Copy Markdown
Collaborator Author

Self-review passed.

I reviewed the complete diff and relevant surrounding code. No actionable issues were found, so no fixes were needed.

Verification:

  • git diff --check — passed.
  • mvn -pl iotdb-core/consensus -am test -Dtest=IoTConsensusRPCServiceProcessorTest -Dsurefire.failIfNoSpecifiedTests=false — passed (2 tests, 0 failures, 0 errors).
  • mvn -pl iotdb-core/datanode -am test -Dtest=TableViewTabletProjectorProviderTest -Dsurefire.failIfNoSpecifiedTests=false — could not complete because the JVM terminated with an insufficient native memory error before producing test results.

@Caideyipi
Caideyipi marked this pull request as draft November 19, 2026 02:27
@Caideyipi
Caideyipi marked this pull request as ready for review November 19, 2026 02:38
…ed-pipe-consensus-subscription

# Conflicts:
#	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/analyze/load/LoadTsFileAnalyzer.java
@Caideyipi Caideyipi changed the title Refactor shared Pipe, consensus and subscription helpers Refactor shared Pipe, consensus, and subscription helpers; cache LOAD models Nov 19, 2026
@Caideyipi
Caideyipi marked this pull request as draft November 19, 2026 03:19
@Caideyipi
Caideyipi marked this pull request as ready for review November 19, 2026 03:23
@Caideyipi
Caideyipi marked this pull request as draft November 20, 2026 09:01
@Caideyipi

Copy link
Copy Markdown
Collaborator Author

CI follow-up for ada233a. The failed run reported five test errors from the renamed dataTablePattern field and four stale LOAD model-cache field references. Updated historical-source tests to use dataTablePattern, LOAD diagnostics tests to seed LoadTsFileModelCache, and projector coverage to assert the TabletBatch API. Resolved the SubscriptionPipeTabletEventBatch conflict by retaining the batch-scoped projector provider and master tablet-size recalculation after filtering. Verification: Spotless apply passed; focused datanode reactor run passed 54 tests with 0 failures and 0 errors; git diff --check passed. The fixes are in reviewed commit ada233a, matching both local and remote PR heads. PR remains Draft pending a fresh independent review of the final diff.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant